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Outcomes-Based Assessment and Learning: Trialling Change in a 
Postgraduate Civil Engineering Course 


Abstract 

This paper aims to demonstrate how assessment tasks can function within an outcomes-based learning 
framework to evaluate student attainment of learning outcomes. An outcomes-based learning framework 
designed to integrate teaching, learning, and assessment activities was developed and implemented in a civil 
engineering master-level course. The assessment instruments for this course were designed together to form a 
deliberate, balanced, and practical approach to evaluating student attainment of learning outcomes within the 
outcomes-based learning initiative. Direct evidence of student learning was derived through analysis of 
student results in assessment tasks constructively aligned with intended outcomes of learning. Student 
feedback provided indirect evidence of student attainment of learning outcomes and confirmed the 
effectiveness of the learning approach implemented in the course under investigation. Results of the direct 
assessment instruments were, generally, consistent with the student self-perception confirming achievement 
of learning outcomes. Students tended, however, to overestimate the level of attainment of learning outcomes. 
Results of the present study are anticipated to assist educators and researchers to efficiently and effectively 
implement and evaluate outcomes-based learning in higher education thus improving educational quality and 
student learning 
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Introduction 

Enhancing educational practice and student learning is an area of importance in higher education 
worldwide, as shown by widespread emphasis on systems of standards and evaluation. The 
Bologna Declaration advocates adoption of meaningful criteria and methodologies for quality 
assurance in European higher education (Bologna Declaration 1999). The Western Association of 
Schools and Colleges (WASC) accreditation agency requires all educational programs seeking 
accreditation at the undergraduate and graduate levels to establish student learning outcomes, 
develop plans for assessments and use the results to enhance student learning (Lindholm 2009; 
WASC 2013). Similarly, the United Arab Emirates (UAE) has a national Qualifications 
Framework (QF Emirates 2012) established by National Qualifications Authority (NQA). The 
QFEmirates provides a frame of reference for the quality of learning qualifications that places 
student outcomes as a core focus. 

The first author’s institution (UAE University) is going through quality-assurance (QA) and 
quality-enhancement (QE) processes that reflect local and global priorities. The institution’s 
approach has required changes not only in the execution of curricula, but also in the underlying 
structure and approaches to student engagement, especially in the area of assessment. Achieving 
deep change in higher-education curricula is, however, notoriously difficult, particularly in the 
area of assessment (Deneen & Boud 2014; Trawler et al. 2003). Thus, analysing and reporting has 
become an essential part of the QA and QE processes. 

The outcome-based learning (OBL) approach has been adopted in the first author’s institution. 
OBL is a student-centred approach that focuses on academic outputs rather than educational inputs 
(Barkley & Major 2016; Biggs & Tang 2007; Carless 2015). It is a well-recognised approach to 
reconstructing curricula to better align learning and assessment with the intended outcomes of 
learning (Arshada et al. 2012; Biggs & Tang 2007; Cox 2009; Hendry 2014; Hughes 2013; Lixun 
2011; Osman et al. 2012; Wahab et al. 2011). Implementing OBL in higher education is a 
challenging task; several interrelated activities pertaining to teaching, learning and assessment 
must be conducted concurrently. Proper constructive alignment between teaching, learning, 
assessment activities and learning outcomes is crucial for successful implementation of the OBL 
approach (Barkley & Major 2016; Biggs & Tang 2007). 

OBL is not without controversy. Critics say it can be perceived and resisted as an external 
imposition that ignores context or stifles innovation, especially in assessment (Deneen & Boud 
2014). Similarly, the adoption of an OBL approach must avoid the significant challenges of 
“conceptual reification and rigidity in implementation” (Ewell 2005, p. 27). The adoption of an 
OBL approach to enhancing and assuring quality must be accompanied by a careful examination 
of the legitimacy of the process, the perceptions of stakeholders, the outcomes and the relationship 
of assessment to the enterprise. 

The aim of this paper is to report findings from a research study on the perceptions and associated 
learning outcomes of students in an OBL trial. An OBL framework designed to integrate teaching, 
learning and assessment activities was developed and implemented in a civil engineering master’s 
level course offered at the first author’s institution in the second semester of the 2014-2015 
academic year. The assessments for this course were designed together to form a deliberate, 
balanced and practical approach to evaluating students’ attainment of learning outcomes within an 
outcomes-based learning framework. The objectives of this study were to determine 1) how the 
assessments function in relationship to an OBL framework and to student achievement, 2) the 
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perceptions of students in the trialled approach and their achievement and 3) whether the approach 
appears to meet desired benchmarks of innovation. 

This paper presents a context for the trial by reviewing salient areas of the literature. The 
institutional context and trial are then presented, along with the means by which data was collected 
and analysed. Results are then presented and their significance is discussed. The results are 
intended to inform higher education educators and researchers seeking to use OBL effectively, 
especially in terms of achieving constructive alignment, fostering positive student response and 
affecting educational quality and assessment practices. 

Literature review 

Enhancing higher education 

QA and QE in higher education are concerns worldwide. QA may be understood as processes of 
reporting and accountability that construct an evidence-based case of an institution meeting 
credible internal and external aims and objectives (Nicolson 2011; US Department of Education 
2006). QE then is the formative means by which these processes and their accompanying 
responses are developed (Nicolson 2011). QE may include a host of activities; one activity 
deemed essential is trialling techniques that improve student learning and better align assessment 
with curricula in terms of supporting and accounting for achievement (Deneen & Boud 2014). 

This study may be seen as fitting within this framework of the relationship of QA and 
QE(henceforth referred to as QA/QE). 

Higher-education institutions and accrediting bodies see outcomes as central to QA/QE processes. 
Learning outcomes provide key benchmarks for maintaining standards, judging educational 
quality and enhancing teaching and learning (Biggs & Tang 2007; Carless 2015; Hughes 2013). In 
the Bologna Process, learning outcomes play a critical role in enhancing and developing 
equivalence in the standards and quality of European qualifications (Bologna Declaration 1999). 
All undergraduate programs seeking accreditation from the Engineering Accreditation Committee 
of ABET (Accreditation Board for Engineering and Technology) must demonstrate that they 
accomplish learning outcomes that prepare graduates to attain their program objectives (ABET 
2014). ABET learning outcomes describe what students are expected to know and be able to do by 
the time of their graduation (ABET 2014). The WASC accreditation process requires both 
undergraduate and graduate programs to document evidence of students’ achievement of intended 
learning outcomes (WASC 2013). 

Such approaches place students’ outcome achievement at the centre of QA/QE. Changing 
educational practice to reflect an enhanced student focus represents a challenge to many 
institutions of higher education. Such changes require thoughtful actions that develop and connect 
learning, teaching, assessment and curriculum (Trowler et al. 2003). Assessment is a critical 
aspect of this, as it provides evidence of student achievement and may be used as part of an 
outcome-oriented QA/QE process (Carless 2015; Ewell 2005). As quality and accreditation 
processes centre more on outcome achievement, assessment must change to meet these demands. 
Achieving change in assessment practices in higher education, however, is challenging, complex 
and difficult to manage, and often meets with resistance (Deneen & Boud 2014). There are many 
reasons for this. Assessment change is difficult to model or sustain past the trial phase of an 
initiative (Trowler et al. 2003). Actual outcomes of such initiatives are often quite different to the 
intended outcomes (MacDonald & Joughin 2009). Staff and students who do not perceive the 
change as an authentic opportunity for enhancement may resist or even subvert the initiative 
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(Bromage 2006; Deneen & Boud 2014; Mclnnis 2006; Trawler & Bamber 2005). This may 
account for a paradox in higher education: while assessment may seem the best area to focus on 
for productive change, it is often the least successful (Mclnnis 2006). 

Trial and study of such changes are therefore warranted. Specifically, these challenges call for a 
trial that focuses on assessment, provides a model or framework for aligning assessment with 
outcomes and learning and takes into account the perspectives of stakeholders involved in the trial. 
The trial reported in this study was developed in response to these imperatives and challenges. 

Outcomes-based approaches to changing curricula 

Barkley and Major (2016) identified three interrelated components for effective pedagogy: (1) 
identifying and communicating clear learning goals and outcomes, (2) helping students achieve 
these outcomes through active/engaged learning and (3) analysing, reporting and reflecting on 
results to improve learning. Implementing these interconnected components in higher education 
necessitates a change in educational practices to focus on what students have to learn rather than 
what educators have to teach. Implementation of OBL requires a shift from educational inputs to 
outputs in the form of direct and/or indirect evidence of student achievement to judge educational 
quality. For the purposes of this study, direct evidence consists of data that directly indicates 
student achievement. Indirect evidence consists of data on students’ perception of achievement. 
Figure 1 summarises an outcomes-based learning framework. Assessment of student learning is 
central in this process because student performance in assessment tasks, constructively aligned 
with intended learning outcomes, can be used to identify what students have learnt and achieved. 



Figure 1 . Outcomes-based learning framework 
An outcomes-based learning approach aligns well with the QF£mi rates and with the three 
components of the effective pedagogy identified by Barkley and Major (2016). OBL approaches at 
institutional, program and course levels often have different, albeit interconnected, goals. For the 
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purposes of this paper, “institution” refers to the whole university; “program” refers to a sequence 
of study ending in a degree (in this case, a master of science in civil engineering); “courses” are 
the specific units of study, consisting of different topics and taught over the period of a semester, 
as part of a program; and “modules” are topic-based learning engagements that the student 
experiences as part of a course. 

An OBL approach at the institutional level is typically used for large-scale evaluation and 
accountability; hence, it often focuses on generic graduate attributes (Biggs & Tang 2007; Carless 
2015). An OBL approach at the program level is established in a way that students can achieve 
specific discipline-oriented program goals. This cascades into learning outcomes at the course 
level. Governments and international and national accreditation agencies require evidence of 
student achievement and learning at the course, program and institutional levels to judge the 
quality of education. 

Successful enactment of OBL in higher education requires constructive alignment between 
teaching, learning activities, assessment tasks and learning outcomes at the course level, proper 
mapping of course learning outcomes to program learning outcomes and reconciliation of 
program-specific learning outcomes with the desired generic graduate attributes at the institutional 
level. There is a risk, though, that this interconnectivity may mutate the function of OBL to be a 
bureaucratic burden rather than a useful tool to promote and improve student learning (Carless 
2015; Carless & Zhou 2016; Deneen & Boud 2014; Hussey & Smith 2008). Similarly, different 
stakeholders may have different perceptions of an OBL initiative, which may affect the experience 
and “buy-in” to the initiative (Deneen et al. 2013). Research is therefore needed to examine 
impact and stakeholder perceptions of a particular OBL approach adopted by an institution. 

Innovative assessment 

Assessment is required in education to demonstrate students’ learning, judge their performance, 
satisfy demands for accountability and, more importantly, support and advance student learning 
(Carless 2015). It should serve formative and summative purposes. Innovative assessments focus 
on the immediate assessment task, but also prepare students for lifelong learning (Boud & 
Falchikov 2006; Boud & Soler 2016; Nguyen & Walke 2016). Innovative assessments may often 
perform double duty: a single task should allow for both formative and summative purposes to be 
fulfilled (Boud & Soler 2016). Such tasks often harness student engagement in the educational 
process, enhance their experience and reflect on their learning (Carless & Zhou 2016; Fisher et al. 
2005; Johnston et al. 2011; Killen 2000; Mitchell & Delaney 2004, Zhou & Deneen 2016). 
Planning for innovative assessment is therefore necessary as part of an OBL approach. Given the 
twin challenges of OBL and assessment change in higher education, it is all the more important to 
build research and analysis into the change initiative, in terms of both objective enhancement of 
the curriculum and relevant stakeholders’ perceptions of the changes. 

Assessment of learning outcomes at the program level should map onto a similar course-level 
process. Instructors typically devote considerable time and effort to delivering course content, 
scoring and analysing student work and determining student grades. Barkley and Major (2016) 
highlighted the main distinction between grades and assessment. 

Grades are given to individual students and are internal to specific class section, while 
assessment is focused on evaluating the achievement of all students enrolled in the course 
(and sometimes all sections of a course) and the data is intended to be shared primarily 
with external stakeholders (Barkley & Major 2016, p. 53). 


http://ro.uow.edu.au/jutlp/voli 4 /issi/io 


4 



El-Maaddawy and Deneen: Outcomes-Based Assessment and Learning 


The lack of alignment between teaching, learning activities, assessment tasks and 
intended learning outcomes is a major obstacle that hinders the use of student grades in assessing 
student achievement of learning outcomes (Rogers 2003; Shanableh 2014). The use of student 
grades to judge student attainment of learning outcomes without paying attention to whether the 
delivered contents and assessment policy are in alignment with intended learning outcomes may 
produce misleading results (Rogers 2003; Shanableh 2014). By contrast, proper implementation of 
OBL and constructive alignment of innovative assessment tasks with learning outcomes may 
allow student grades to serve as defensible evidence of student learning. 

This paper reports findings from a study of an assessment innovation initiated as part of a larger 
OBL-oriented change. An OBL framework designed to integrate teaching, learning and 
assessment activities was developed and implemented at the first author’s institution. New 
assessment tasks were developed and implemented to engage students in innovative practices in 
alignment with outcomes-based learning changes, and to represent what students would produce 
as an outcome of their study. Students were engaged in project-based assessment tasks designed 
not only to evaluate and demonstrate their learning but also to prepare them for what they might 
encounter in future settings. The assessment innovation reflected identified characteristics of 
successful assessment change. 

Context of the study 

In 2010, within the UAE, Federal Decree No. 1 “Establish and maintain the National 
Qualifications Authority” (NQA) was issued. The NQA approved the qualification framework for 
the UAE, known as the QF Emirates in 2012. All institutions providing higher education in the 
UAE are expected to align their credentials (certificates, diplomas and degrees) with the 
QF Emirates. The Commission for Academic Accreditation (CAA) is charged with monitoring 
compliance with the provisions of QF Emirates and international standards. This is accomplished 
through licensure of higher-education institutions and accreditation of individual programs. 

National qualifications authority/qualification framework of the Emirates 

The QF Emirates defines qualifications based on learning outcomes rather than content or time 
spent on a program (QF Emirates 2012). The focus is on description and achievement of outcomes 
that reflect level-specific qualifications, from the most simple to the most advanced levels of 
learning. The QF Emirates describes the learning outcomes in terms of knowledge, skills, and 
aspects of competence. Aspects of competence are expressed in terms of autonomy and 
responsibility, role in context and self-development. Thus, the QF Emirates framework encourages 
an OBL approach that operates at institution, program and course levels. 

The master of science program in civil engineering 

Civil engineering is designated a priority educational area in the UAE. Rapid development of the 
UAE has placed increasing demands for development of infrastructure, transportation networks 
and both residential and industrial complexes. In response, the department of Civil and 
Environmental Engineering (CEE) at the first author’s institution (UAE University) established a 
master of science (M.Sc.) graduate program in civil engineering in 2007. The first author was the 
director/coordinator of the program and the chair of the graduate studies committee of the CEE 
department from 2011 to 2015. 

As a response to QF Emirates, UAE University began trialing an OBL approach in postgraduate 
education. The M.Sc. graduate program began shifting its focus to student achievement outcomes. 
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Initially, a set of program learning outcomes (PLOs) were developed in alignment with 
QFEmirates Handbook (Table 1). The PLOs are broad in scope, as they specify knowledge, skills 
and competencies that students are expected to achieve by successful completion of the program. 


Table 1 . Learning outcomes of the M.Sc. program in civil engineering at UAE University 


Outcome 

Description 

PLOl 

Describe highly specialised civil-engineering principles, concepts and 
methodologies. 

PL02 

Evaluate the performance of advanced civil-engineering systems and components 
through the use of applicable research principles, analytical methods and 
modelling techniques. 

POL3 

Conduct advanced applied research to develop innovative solutions for highly 
complex civil-engineering problems through the use of appropriately selected 
research methodologies and modern engineering tools. 

PL04 

Apply advanced multidisciplinary problem-solving approaches to critically 
analyse contemporary, sophisticated and highly complex civil engineering 
problems. 

PL05 

Present and critique highly complex civil-engineering issues and communicate 
effectively at a high level of proficiency. 

PL06 

Lead professional activities and manage ethical issues in highly complex civil- 
engineering projects. 

PL07 

Implement the social, environmental, ethical, economic and commercial aspects to 
develop valid decisions affecting highly complex civil-engineering projects. 


OBL intervention in the postgraduate course CIVL 616 

The course Rehabilitation of Structures - CIVL 616 is a master’s level, three-credit-hour elective 
graduate-level course offered by the CEE Department at UAE University. In previous years, a 
traditional input-oriented model focusing on content coverage had been adopted in the course. 
Using the new PLOs, the course was redesigned to focus on outcomes rather than inputs. 
Development of course learning outcomes (CLOs) served as a starting point to shift the 
understanding of quality towards student achievement rather than the instructor’s content coverage 
(Table 2). The CLOs align with the PLOs but are narrower in scope. Each outcome is observable, 
measurable and capable of being understood by students, faculty, external agencies and 
stakeholders. Since CIVL 616 is an advanced course offered to graduate students, there is an 
increased emphasis on application of knowledge and advanced development of skills and 
competencies. 

The shift to an OBL approach necessitates a significant change in assessment (Table 3). 
Assessment tasks have been diversified to emphasise student engagement in the learning process 
and provide increased opportunities for demonstrating attainment of course learning outcomes. 

The new assessment tasks include laboratory projects and research papers; these are designed to 
support and determine within-course achievement as well as to support students’ attainment of 
skills that might be required of a civil engineer beyond the course, in keeping with principles of 
sustainable assessment (Boud & Soler 2016; Boud & Falchikov 2006; Nguyen & Walke 2016). 


Table 2. Comparing traditional course objectives with the new course learning outcomes 


Traditional course outline 

OBL course outline 

Course objectives: 

Course learning outcomes: 
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1. Introduce students to causes of 
concrete deterioration and damage 
mechanisms. 

2. Familiarise students with evaluation 
techniques for structural condition 
assessment. 

3. Introduce students to methods of 
repair and rehabilitation of existing 
structures. 

4. Introduce students to principles of 
structural strengthening using 
advanced composites. 


Upon completion of the course, students should 
be able to: 

CLOl. Describe damage mechanisms and 
principles of structural strengthening. 

CL02. Identify causes of defects, cracks, damage 
and deterioration of concrete structures. 

CL03. Develop an appropriate repair strategy for 
a deficient structure, taking into consideration the 
social, economic and commercial aspects. 

CL04. Perform analysis and design of reinforced 
concrete elements strengthened with advanced 
composites, using appropriate problem-solving 
approaches and international code provisions. 
CLO 5. Conduct experiments for condition 
assessment, corrosion monitoring and 
strengthening of columns using advanced 
techniques. 

CL06. Report findings and critique recent 
research on assessment and rehabilitation of 
structures. 

CLO 7. Communicate effectively with peers and 
clients at a high level of proficiency. _ 


Table 3. Comparison of assessment tasks in the old course outline and new OBL course outline 


Assessment tasks in the old course outline 

Assessment tasks in the new OBL course outline 

1. Mid-term exam (30%)° 

2. Final exam (30%) fl 

3. Three assignments (40%)" 

1. Exam 1 (25%) a 

2. Exam 2 (25%) a 

3. Assignment 1 (10%)* 

4. Assignment 2 (10%)* 

5. Laboratory project written report and 
presentation (15%) c 

6. Research paper and presentation (15%) e 


" Individual assessment. 
b Group assessment. 

1 Group assessment for the written document and individual assessment for the presentations. 


The laboratory project aimed to provide students with hands-on training and to serve as a learning 
activity. Students were required to conduct testing, use analytical approaches for performance 
evaluation, make comparisons with predictions of international guidelines and standards, analyse 
data and report results. For the research paper, students reviewed and discussed original and recent 
journal articles, describing a major scientific advancement in a research area related to course 
topics. Students made presentations, submitted reports and participated in discussions. The 
laboratory project and research paper were used as learning activities and assessment tasks, since 
they required the development of new knowledge, skills and dispositions or extension of those 
introduced in the undergraduate studies. Sample hands-on learning activities conducted during the 
laboratory project are shown in Figure 2. 
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Corrosion monitoring test 


Bulk resistivity test 


Ultrasonic pulse velocity test 


Figure 2. Sample hands-on learning activities conducted during the laboratory project 


Students worked in groups in the homework assignments, laboratory project and research paper to 
foster collaborative investigation. For the laboratory project and research paper, each group 
submitted a written document and delivered an oral presentation; this was to enhance students’ 
technical writing and communication skills. Two exams were included in the assessment plan to 
give students the opportunity to demonstrate individual achievement (Killen 2000). 


Successful implementation of OBL requires proper mapping between CLOs and PLOs and 
“constructive alignment” between teaching, learning activities, assessment tasks and intended 
learning outcomes (Biggs & Tang 2007). The assessment tasks adopted in this course were 
“constructively aligned” with intended CLOs that were mapped to specific PLOs (Table 4). This 
indicates that the course of the current study has been developed as a legitimate OBL approach. 


Table 4. Alignment between CLOs, PLOs and assessment tasks 


CLOs 

Program learning outcomes (PLOs) 

Assessment task" 

PLOl 

PL02 

PL03 

PL04 

l/"> 

o 

CL. 

so 

o 

J 

Oh 

PL07 

CLOl 

X 







HW1, HW2, EMI, EM2 

CL02 

X 







HW1, EMI 

CL03 

X 

X 


X 



X 

HW1, EMI 

CL04 

X 

X 


X 




HW2, EM2, LR 

CL05 


X 

X 





LR, LP 

CL06 



X 


X 

X 


LR, LP, RR, RP 

CL07 





X 

X 


LP, RP 


a HW = homework, EM = Exam, LR = Lab report, LP = Lab presentation, RR = Research paper report, RP = Research 
paper presentation. 
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Methodology 

The research addresses the questions: (1) How can assessment tasks function within an OBL 
framework to evaluate student attainment of learning outcomes? and (2) What does direct and 
indirect evidence indicate regarding the effectiveness of an OBL approach that uses innovative 
assessment? 

Participants 

The CIVL 616 course under investigation included 12 master’s students. The course was delivered 
by the first author. Evidence of student achievement of course learning outcomes was collected 
through three types of data: one direct and two indirect (Table 5). Student participation in the 
surveys was voluntary. Surveys were distributed at the end of the course. Appropriate protocols 
for maintaining anonymity were observed. The student self-assessment survey of course outcomes 
was distributed and collected by the first author. It did not include any questions related to student 
identity (such as student’s name or ID number). The online student course evaluation survey was 
administered by the Planning, Academic and Institutional Development Department (PAIDD) at 
UAE University. PAIDD maintained confidentiality of student identity. Since the surveys 
employed in the current study were anonymous to the authors, there was no risk of coercion. 
Results of the current study are reported only as aggregate data to maintain participants’ 
anonymity and confidentiality. 


Table 5. Direct and indirect evidence of student learning 


Assessment task 

Direct/indirect 

Focus 

Two exams 

Two homework assignments 
Laboratory project 

Research paper 

Direct 

Actual student achievement of course 
learning outcomes 

Student self-assessment survey of 
course outcomes 

Indirect 

Student self-perception of achievement 
of course learning outcomes 

Online student course evaluation 
survey 

Indirect 

Effectiveness of the learning approach 
Alignment between the teaching, 
learning activities, assessment tasks and 
intended learning outcomes 

Value of the course from student 
perspective 


Data collection 

Students’ work in the direct-assessment tasks was evaluated by the first author using criterion- 
referenced assessment. Direct evidence was derived through analysis of student results in the 
exams, laboratory project, research paper and homework assignments. The homework 
assignments, exams and laboratory project were carefully designed to directly measure student 
attainment of CLOl to CL04 (Table 4). The laboratory project was also used along with the 
research paper in assessing student attainment of CL05 to CL07 (Table 4). 

The indirect measures included a student self-assessment survey of course outcomes and a 
standard online student course evaluation survey (Table 5). Students completed the surveys by the 
end of the semester before the final assessment task was handed back; hence, their opinion was 
based on an incomplete picture of the grade they would get. In the student self-assessment survey 
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of course outcomes, a customised questionnaire was used to obtain students’ perceptions of course 
learning outcome achievement. Students were asked to rate their level of achievement of each 
course learning outcome from very low to very high on a five-point Likert-type scale. 

Although the standard online student course evaluation survey is general in approach, it includes 
key items that can be used to assess the effectiveness of an OBL learning approach and the 
alignment between the learning activities, assessment tasks and intended learning outcomes. For 
the purposes of this paper, the word “objectives” included in some of the survey items was 
replaced by the word “outcomes”. The key survey items were used to explore students’ 
perceptions of the value of the OBL aspects of the course. Students were asked to indicate the 
degree of their agreement with each statement using a five-point Likert-type scale. Six survey 
items were relevant to the current study: 

1. The course [outcomes] were clearly explained. 

2. There was close agreement between the stated course [outcomes] and what was actually 
covered. 

3. Evaluation methods were clearly explained. 

4. The assignments in the course were clearly related to the course [outcomes], 

5. The general climate in this course was good for learning. 

6. The course added to my knowledge in my major. 

It is possible in the future to break item 4 of the online student course evaluation survey into 
multiple items corresponding to specific assessment tasks. The decision not to do so in this 
iteration of the research reflects the balance researchers must always strike between increasing 
how fine-grained an instrument is and engendering “survey fatigue” among users. A “per-task” 
evaluation could be undertaken in future studies, which may yield a finer-grained picture of course 
assessment. 

Approach to analysis 

Actual student performance : Student performance, indicated by their average earned grades in the 
direct-assessment tasks linked to each CLO, was used as a direct evidence of student learning. 
Accordingly, student attainment of a specific course learning outcome was calculated by 
averaging student scores in all assessment tasks linked to the CLO in question (Eq. 1). The 
attainment level of each course learning outcome expressed as a percentage, AL„ can be 
represented on a five-point scale using Eq. 2 to obtain the attainment-level weight value in the 
range of 1 to 5; this allows results of actual student performance to be compared to students’ 
perception of outcome achievement. A course learning outcome i was considered achieved when 
the corresponding student attainment level based on actual student performance, A L,. was in the 
range of 70% to 80% (i.e. 3.5 < ALWi < 4). For student attainment level of AL, > 80% (i.e. ALW-, > 
4), the CLO was considered adequately achieved. 


AL i =-f j SR J 

n j=i 

Eq. (1) 

ALWi J^l\ 5 ) 

' 1 100 r ’ 

Eq. (2) 


AL, = attainment level of course learning outcome i based on actual student performance (%). 
SR: = mean value of student scores in assessment task j linked to course learning outcome i (%). 
n = number of assessment tasks linked to course learning outcome i. 


http://ro.uow.edu.au/jutlp/voli 4 /issi/io 


10 



El-Maaddawy and Deneen: Outcomes-Based Assessment and Learning 


ALWi = attainment level weight value of course learning outcome i based on actual student 
performance (1 to 5). 

Student self-perception : The consistency between the actual student performance and student self¬ 
perception for a specific course learning outcome i is estimated by calculating a corresponding 
consistency index /,. The consistency index between the student self-perception and actual student 
performance for each course learning outcome is calculated by dividing the corresponding average 
score obtained from the student self-assessment survey of course outcomes by the corresponding 
attainment-level weight value (Eq. 3). Index values were considered indicative of consistency 
between results when the error band was in the range of 10% (i.e. 0.9 < /, < 1.1). 


/. - 


M. 


ALW ; 


Eq. (3) 


/, = consistency index between student self-perception and actual student performance for course 
learning outcome i. 

Mi = mean score pertaining to course learning outcome i obtained from the student self-assessment 
survey of course outcomes. 

Effectiveness of the learning approach : Key items of the standard online student course evaluation 
survey that were related to the effectiveness of the learning approach have been used to reflect 
student perspectives about the course delivery. Results of the key items have been used to examine 
how students perceived the course under investigation, which had been designed according to the 
OBL approach, and to identify whether the teaching, learning activities and assessment tasks were 
properly aligned with intended course learning outcomes from the students’ perspective. A 
standard of 80% agreement (“agree” plus “strongly agree”) or more and a mean score of 4 or more 
in each item were considered as indicating successful implementation of the OBL approach in the 
course under investigation. 

Limitations 

Although the student enrollment in the course under investigation can be considered relatively 
high for a postgraduate engineering course, 12 students are still a small number of participants 
from a statistical perspective. The variety of direct and indirect data sources used in the current 
study are meant to offset this limitation. Although the assessment approach introduced and 
implemented in the current study is independent of the sample size, further research is needed to 
confirm its applicability and practicality for classes with higher enrollment. Due to the recognised 
difficulties and complexity of assessment change, however, starting a trial at the current scope was 
deemed useful and appropriate (Carless & Zhou 2016). Suggestions for expanding the scope of 
inquiry are addressed in the conclusion section. 

Results 

Actual student performance 

The attainment levels of CLOs based on the actual student performance are summarised in Table 
6. The table shows that all CLOs were adequately achieved. The attainment level of all CLOs, 
based on student performance, was on average 85%, with a standard deviation of 2% and 
coefficient of variation of 2.5%. The highest attainment level of 87.4% was recorded for CL02 


n 
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and CL03 (student knowledge and skills, respectively). On the other hand, CL07, pertaining to 
student communication skills, exhibited the lowest attainment level, 81.7%. One interpretation is 
that the use of presentations (the mode of achieving CL07) should be enhanced or increased. 


Table 6. Assessment results based on actual student performance 


CLOs 

Attainment level based on actual student 
performance 

Level of achievement of CLOs 
based on actual student performance 

Percent 

(AL) 

Weight 

(ALW) 

CLOI 

86.6 

4.33 

Adequately achieved 

CL02 

87.4 

4.37 

Adequately achieved 

COL3 

87.4 

4.37 

Adequately achieved 

CL04 

85.9 

4.30 

Adequately achieved 

CL05 

83.5 

4.18 

Adequately achieved 

CL06 

85.2 

4.26 

Adequately achieved 

CL07 

81.7 

4.08 

Adequately achieved 

Average 

85 

4.27 


STDEV 

2 

0.11 


COV (%) 

2.5 

2.5 



Consistency index 

Figure 3 compares the attainment level weight values of CLOs, based on actual student 
performance with results of student self-perception, obtained from the sUident self-assessment 
survey of course outcomes. The corresponding consistency indices are given in Table 7. While 
results of student self-perception were generally consistent with actual student performance, 
students tended to overestimate the attainment level of intended course learning outcomes. This 
was more evident for CL07, with a student consistency index value of 1.07. 


I Student self-perception 


□ Actual student performance 


® 3 


c 2 


< 




0 


CLOI CL02 CL03 CL04 CLOS CL06 CL07 


Course learning outcome 

Figure 3. A comparison between actual student performance and student self-perception 
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Table 7. Consistency indices 


CLOs 

Consistency index of student self-perception 

Index (I) 

Observation 

CLOl 

0.99 

Consistent 

CL02 

1.06 

Consistent 

COL3 

1.00 

Consistent 

CL04 

0.97 

Consistent 

CL05 

1.04 

Consistent 

CL06 

1.02 

Consistent 

CL07 

1.07 

Consistent 


Students’ judgements about their achievement depend on their level of understanding of 
assessment requirements and how their performance would be evaluated against these 
requirements (Wesp et al., 1996). That student and instructor judgements of student performance 
tend to vary is well-established (Boud & Falchikov 1989; Brown & Harris 2014). Students in this 
study overestimated their level of attainment of course learning outcomes possibly because they 
were not given an opportunity to evaluate their own work against reference criteria/standards. 
Involvement of students in the application of a marking rubric to a sample assessment output 
could result in a better consistency between student and instructor assessment. Providing students 
a range of exemplars of high-, medium- and low-level student work may help students to better 
understand the requirements of assessment (Cowan 2002, 2006). Engagement of students in self- 
and/or peer-assessment tasks may improve their self-evaluative capacity and expertise (Carless 
2015). Closing this judgment gap and fostering in students the capacity to more accurately judge 
their own work would have the additional benefit of developing a recognised sustainable 
competency through engagement with assessment (Boud & Soler 2016). 

Online student course evaluation survey 

The mean scores of the key items of the online student course evaluation survey and frequency 
charts are shown in Figures 4 and 5, respectively. All items achieved an agreement level (“agree” 
plus “strongly agree”) greater than 80% and a mean score > 4, which confirmed students’ 
perception of successful implementation of the OBL in delivering the course under investigation. 



Survey item 

Figure 4. Mean scores of the key items of the online student course evaluation survey 
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1. The course [outcomes] were clearly explained. 2. There was close agreement between the stated 

course [outcomes] and what was actually 
covered. 



3. Evaluation methods were clearly explained. 



4. The assignments in the course were clearly 
related to the course [outcomes]. 



5. The general climate in this course was good 
for learning. 



6. The course added to my knowledge in my 
major 


Figure 5. Frequency charts of the key items of the online student course evaluation survey 


Discussion 

A key finding was the confirmation of the feasibility of adopting OBL in a postgraduate civil- 
engineering course. A related finding was the validation of a revised and innovative assessment 
approach for promoting and determining outcome achievement. Findings demonstrated how 
assessment tasks can function within an OBL framework and meet benchmarks of innovation. 
These include performing the double duty of developing and demonstrating achievement, as well 
as the twin purposes of addressing immediate learning priorities while still developing graduate 
competencies (Boud & Falchikov 2006; Boud & Soler 2016). One implication is that an OBL 
initiative and assessment innovation may compliment each other when implemented together, as 
they may provide a balance of priorities while reducing the threat of reification through OBL 
(Ewell 2005) and resistance to assessment change (Deneen & Boud 2014). 
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Results also provided insight into areas for improving learning engagement and assessment. The 
student feedback to item 6 of the online student course evaluation survey indicated that students 
appreciated the value of the course under investigation. All students agreed that the course added 
to their knowledge in their major. This confirmed the concept of using learning as development 
(Ewell 2005). Although the attainment level of CL07, pertaining to student communication skills, 
was above 80%, it was the lowest of all elements. This suggests that in future semesters, students 
should deliver multiple presentations throughout the delivery of the course to further improve their 
communication skills. Although the agreement levels and mean scores of items 3 and 5 of the 
online student course evaluation survey were acceptable, about 17% of the students were uncertain 
about the clarity of the evaluation methods and the quality of the learning environment. More 
attention should then be given to these items in future semesters. Evaluation methods should be 
repeatedly explained to students throughout the delivery of the course. This validates the idea that 
undertaking an OBL and/or assessment initiative should be accompanied by a component 
requiring diligent research (Deneen et al. 2013; Deneen & Boud 2014). This can provide insight 
into ongoing improvement, and may therefore play a significant role in properly positioning OBL 
and assessment innovation within a broader QA/QE context. 

Conclusion 

Results of the current study confirmed the feasibility of adopting OBL in a postgraduate civil- 
engineering course when accompanied by innovative assessment practices. The implemented 
assessment practices provided support for and evidence of student learning. This strongly suggests 
that there was a valuable engagement. A constructive alignment of the analysis of student results 
in assessment tasks with intended outcomes provided direct evidence of student attainment of 
learning outcomes. Students’ perceptions of their own attainment of learning outcomes was 
generally consistent with their actual performance in the direct-assessment tasks. Only the results 
of item 6 of the online student course evaluation survey could indicate whether value-added 
learning occurred. From a quality-assurance standpoint, both attainment and development may be 
important. This study contributes to research suggesting that evaluation of students should be 
crafted more specifically to the frameworks and intentions of change and innovation (Deneen et al. 
2013). Using fine-tuned instruments more sensitive to OBL and assessment innovation, future 
studies might produce more varied and differentiated evidence for the impact of these innovations 
in a higher-education context. 

Similarly, gathering evidence of students’ perceptions of the OBL experience and their 
achievement proved meaningful. First, it demonstrated that students tended to overestimate their 
level of attainment of learning outcomes. Closing this gap is an area of extraordinary importance 
to the development of sustainable assessment practices by institutions and sustainable 
competencies in students (Boud & Soler 2016). Thus, it would be productive to pursue this point 
through further trials within the context of developing an OBL approach to assessment. Second, 
this demonstrates the importance of collecting perceptual data as part of an assessment-change 
initiative. Stakeholder perceptions matter in terms of the success of assessment initiatives 
(Bromage 2006; Deneen & Boud 2014; Mclnnis 2006; Trawler & Bamber 2005). As this initiative 
expands to include other instructors and disciplines, this specific line of inquiry should be 
expanded to include student and instructor perceptions. 
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